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ABSTRACT 

This report presents findings from an evaluation of 
Philadelphia’s Children Achieving reform agenda, which was designed to 
improve the Philadelphia public schools. Between 1995-1996 and 2000-2001, 
interviews were conducted with hundreds of teachers, principals, parents, 
students, district officials, and civic leaders. Researchers sat in on 
meetings where the plan was developed, observed its implementation in 
classrooms and schools, conducted two systemwide surveys of teachers, and 
carried out independent analyses of the district’s test results and other 
indicators of system performance. Results indicate that Children Achieving 
reforms are related to students’ school achievement. Children Achieving 
reform components demonstrate significant relationships to fourth grade 
reading achievement. Schools with more obstacles to learning and with more 
safety concerns have a higher probability that fourth graders will score 
below the mean in standardized reading achievement. These relationships are 
even higher among high poverty schools. Well-implemented small learning 
communities provide a good learning environment and are better implemented in 
safer schools with a greater sense of teacher professional community and 
fewer obstacles to learning. School size and teacher experience relate to 
both obstacles to student learning and teacher-reported professional 
community. Three appendices present analysis of the Survey of Philadelphia 
Teachers, the 1998-1999 Teacher Survey, and rationale for and description of 
variable categories. (SM) 
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ABOUT THE 
CHILDREN 
ACHIEVING 
CHALLENGE 

In February 1995, shortly after the 
School Board of Philadelphia adopted 
Children Achieving as a systemic reform 
agenda to improve the Philadelphia 
public schools, the Annenberg 
Foundation designated Philadelphia as 
one of a few American cities to receive 
a five-year $50 million Annenberg 
Challenge grant to improve public 
education. 

Among the conditions for receiving the 
grant was a requirement to raise two 
matching dollars ($1 00 million over five 
years) for each one received from the 
Annenberg Foundation and to create 
an independent management structure 
to provide program, fiscal, and 
evaluation oversight of the grant. In 
Philadelphia, a business organization. 
Greater Philadelphia First, assumed this 
responsibility, and with it, the challenge 
of building and sustaining civic support 
for the improvement of public 
education in the city. 

Philadelphia's Children Achieving was a 
sweeping systemic reform initiative. 
Systemic reform eschews a school-by- 
school approach to reform and relies on 
coherent policy, improved coordination 
of resources and services, content and 
performance standards, decentral- 
ization of decision-making, and 
accountability mechanisms to transform 
entire school systems. Led by a dynamic 
superintendent and central office 
personnel. Children Achieving was the 



first attempt by an urban district to test 
systemic reform in practice. 

EVALUATION OF 

CHILDREN 

ACHIEVING 

In 1996, the Consortium for Policy 
Research in Education (CPRE) at the 
University of Pennsylvania and its 
partner. Research for Action (RFA), 
were charged by the Children 
Achieving Challenge with the 
evaluation of Children Achieving. 
Between the 1995-1996 and 2000-2001 
school years, CPRE and RFA 
researchers interviewed hundreds of 
teachers, principals, parents, students. 
District officials, and civic leaders; sat in 
on meetings where the plan was 
designed, debated, and revised; 
observed its implementation in 
classrooms and schools; conducted two 
system-wide surveys of teachers; and 
carried out independent analyses of the 
District's test results and other 
indicators of system performance. An 
outline of the research methods used 
by CPRE and RFA is included in this 
report. A listing of the reports on 
Children Achieving currently available 
from CPRE is found below. There will 
be several additional reports released 
in the coming months. New reports will 
be listed and available as they are 
released on the CPRE web site at 
www.cpre.org. 
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CHILDREN 
ACHIEVING'S 
THEORY OF ACTION 

To assess the progress and effects of a 
comprehensive reform such as Children 
Achieving, it is essential to understand 
its "theory of action," that is, the 
assumptions made about what actions 
or behaviors will produce the desired 
effects. A summary of the Children 
Achieving theory of action follows: 

Given high academic standards and 
strong incentives to focus their efforts 
and resources; more control over 
school resource allocations, 
organization, policies, and programs; 
adequate funding and resources; more 
hands-on leadership and high-quality 
support; better coordination of 
resources and programs; schools 
restructured to support good teaching 
and encourage improvement of 
practice; rich professional development 
of their own choosing; and increased 
public understanding and support; the 
teachers and administrators of the 
Philadelphia schools will develop, 
adopt, or adapt instructional 
technologies and patterns of behavior 
that will help all children reach the 
District's high standards. 



ADDITIONAL 
READING ON 
CHILDREN 
ACHIEVING 

The following publications on the 
evaluation of Children Achieving are 
currently available through CPRE at 
(215) 573-0700, or email your requests 
to cpre@gse.upenn.edu. 

• Recruiting and Retaining Teachers: 
Keys to Improving the Philadelphia 
Public Schools (May 2001) 

• School Leadership and Reform: 

Case Studies of Philadelphia 
Principals (May 2001) 

• Contradictions and Control in 
Systemic Reform: The Ascendancy 
of the Central Office in Philadelphia 
Schools (August 2001) 

• Clients, Consumers, or 
Collaborators? Parents and their 
Roles in School Reform During 
Children Achieving, 1995-2000 
(August 2001) 

• Powerful Ideas, Modest Gains: Five 
Years of Systemic Reform in 
Philadelphia Middle Schools 
(December 2001) 

• An Analysis of the Effect of Children 
Achieving on Student Achievement 
in Philadelphia Elementary Schools 
(February 2002) 
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AUTHORS' NOTE 

The research reported herein was 
conducted by the Consortium for Policy 
Research in Education and Research for 
Action. Funding for this work was 
provided by Greater Philadelphia First 
and The Pew Charitable Trusts. 

Opinions expressed in this report are 
those of the authors, and do not 
necessarily reflect the views of Greater 
Philadelphia First, The Pew Charitable 
Trusts, or the institutional partners of 
CPRE. 

CHILDREN 

ACHIEVING 

EVALUATION 

1995 - 2001 : 

RESEARCH METHODS 

During the past five years, the 
Consortium for Policy Research in 
Education and Research for Action used 
the research methods indicated below 
in their evaluation of the Children 
Achieving Challenge. 

1. 1996-2000 school-level data on 
indicators that made up the 
District's Performance Responsibility 
Index including student scores on 
the SAT-9, student promotion and 
graduation rates, student 
attendance, and teacher 
attendance. 

2. Two census surveys of teachers, the 
first in 1997 and the second in 1999. 
Teachers were asked about reform 
implementation, school conditions. 



and teaching practices. There was a 
greater than 60 percent response 
rate on both surveys. 

3. School indicators describing teacher 
and student characteristics in 1996 
and 1999 obtained from the School 
District of Philadelphia's Information 
Services. These data included school 
enrollment, number of teachers, the 
proportion of students qualifying for 
free or reduced price lunch, among 
other indicators. These data were 
used for descriptive purposes and in 
hierarchical linear and logistic 
regression models to help 
understand the relationships among 
reform implementation, student 
outcomes, and school 
characteristics. 

4. Five years (1995-1996 through 1999- 
2000) of qualitative research in 49 
schools (26 elementary, 11 middle, 
and 12 high schools) in 14 clusters. 
Qualitative research included: 
interviews of teachers, principals, 
parents, outside partners who 
worked in the schools, and in a few 
cases, students; observations of 
classrooms, SLC meetings, 
professional development sessions, 
and school leadership team 
meetings; review of school 
documents (School Improvement 
Plan, budget, etc.); and intensive, 
multi-year case study research in a 
subset of 25 schools (13 elementary, 
five middle, and seven high schools). 

5. Interviews of central office and 
cluster staff and observations of 
meetings and other events. 
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6. Interviews of 40 Philadelphia civic 
leaders (including political leaders, 
leaders in the funding community, 
public education advocates, 
journalists, and business leaders). 

In addition, numerous other studies 
conducted during Children Achieving 
informed this evaluation. These 
included: Bruce Wilson and Dick 
Corbett's three-year interview study of 
middle school students; an evaluation 
of the Philadelphia Urban Systemic 
Initiative in Mathematics and Science 
conducted by Research for Action; the 
Philadelphia Education Longitudinal 
Study conducted by Frank Furstenberg 
at the University of Pennsylvania; and 
the evaluation of the William Penn 
Foundation's initiative in two clusters, 
conducted by the National Center for 
Restructuring Education, Schools, and 
Teaching. 
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INTRODUCTION 

E vidence over the last few 

decades clearly indicates that 
America's city schools need 
educational improvement. 
Specifically, United States urban 
education systems are characterized by 
levels of low educational achievement, 
high dropout rates, and too many 
graduates who are inadequately 
prepared for college or employment. 

With the number of children in poverty 
rising,’ documentation that high- 
poverty schools are disproportionately 
composed of ethnic and racial 
minorities,^ and evidence that students 
with the lowest levels of academic 
achievement are more likely to be 
found in high-poverty schools,^ 
researchers and policymakers have 
identified urban minority children as 
especially vulnerable to educational 
failure. 

Now more than ever, poor school 
performance can have potentially 
devastating consequences for poor 
children. While a lack of education has 



' National Center for Education Statistics, Urban 
schools: The challenge of location and poverty. 
Washington, D.C.; U.S. Department of Education, 
Office of Educational Research and Improvement, 
1996. F.C. Jones-Wilson, "Alleviating the force of 
poverty on poor minority children." Early Child 
Development and Care 73 (1991), pp. 103-120. 

^ S.W. Duncan, "Families and neighbors as sources of 
disadvantage in the schooling decisions of White and 
Black adolescents." American Journal of Education 
103 (1994), pp. 20-53. A.M. Garibaldi, "Four decades 
of progress and decline; An assessment of African 
American educational attainment." Journal of Negro 
Education 66 (1997), pp. 105-120. 

’ National Center for Education Statistics, Urban 
schools: The challenge of location and poverty. 
Garibaldi, "Four decades of progress and decline." 



been linked for decades to economic 
difficulty and unemployment, experts 
indicate that the consequences of 
educational failure will be increasingly 
more costly as we begin the 21st 
century Evidence to support this claim 
includes the increasing competitiveness 
of the international economy, the loss 
of unskilled and semi-skilled jobs to off- 
shore locations, our society's ever- 
increasing dependence on technology, 
as well as reports documenting the 
requirements of higher educational 
standards for more and more jobs. ^ 

In response to reports insisting on 
improvements in public education, 
educators and policymakers have 
worked together to raise education 
standards in the states.* In an effort to 
reach the new standards, reform efforts 
have been designed and implemented 
across the country to raise children's 
school performance. One approach to 
reform has been to set standards, 
design new assessments and 
accountability procedures aligned with 
the standards, invest in supports to 
improve curriculum and instruction, and 
decentralize more authority to schools. 



“ J. Belsky and C. McKinnon, "Transition to school; 
Developmental trajectories and school experiences." 
Early Education and Development 5 (1994), pp. 106- 
119. National Center for Education 5tatistics, The 
condition of education 1996. Washington, D.C.; U.5. 
Department of Education, Office of Educational 
Research and Improvement, 1996. 

' 5.A. Rush and P.A. Vitale, "Analysis for determining 
factors that place elementary students at risk." 
Journal of Educational Research 87 (1995), pp. 325- 
333. 

‘ U.5. Department of Education, National education 
goa/s. Washington, D.C.; Author, 1992. National 
Education Goal Panel, The national education goals 
report: Building a nation of learners. Washington, 
D.C.; Author, 1996. 
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This strategy is known as systemic 
reform. 

In Philadelphia, the comprehensive 
school reform created to assess and 
address educational needs was called 
Children Achieving. Adopted by the 
School District of Philadelphia in 1995, 
it was one of the most ambitious 
attempts to implement systemic reform 
in an urban education system. 

QUANTITATIVE 
RESEARCH DESIGN 
AND METHODOLOGY 

Children Achieving was based on the 
premise that children can achieve if 
provided with effective learning 
opportunities. Toward this end, the 
Children Achieving reform was 
composed of a series of components 
geared toward actualizing this goal. 
This report provides a detailed 
summary of the quantitative analysis 
conducted as part of Children 
Achieving. 

PHILADELPHIA SCHOOL 
DISTRICT PROFILE 

The School District of Philadelphia is 
the seventh largest district in the 
nation, and during the 2000-2001 
school year, served 208,170 children. 
The District was then composed of 264 
schools organized into 22 clusters. It is 
both racially and ethnically diverse with 
a student population 65.1 percent 
African American, 4.8 percent Asian 
American, 12.6 percent Hispanic, 0.2 
percent Native American, and 17.3 



percent White. Additionally, over 80 
percent of the children served come 
from low-income families, as indicated 
by the number of families receiving Aid 
for Department Children or food 
stamps services. 

QUANTITATIVE ANALYSIS 
TEAM 

When conducting large-scale program 
evaluations, a combination of 
quantitative and qualitative methods 
should be used to ascertain the most 
comprehensive and accurate 
information.^ From the outset, the 
evaluation of Children Achieving was 
designed to utilize both qualitative and 
quantitative methods. Toward that end, 
teams of professionals representing 
qualitative and quantitative 
perspectives were created, and the 
members of both teams worked closely 
together to create an effective and 
comprehensive research effort. 

The quantitative analysis team had two 
key responsibilities. First, quantitative 
support was provided in the planning 
stages of the project. Quantitative 
research methodologists were 
consulted and involved in 
conceptualizing what reform 
components should be considered in 
the research design, and in refining 
how they should be developed in an 
effort to minimize threats to both 
internal and external validity.® Second, 
quantitative research staff contributed 

^ R.F. Boruch, Randomized experiments for planning 
and evaluation: A practical guide. London: Sage 
Publications, 1997. 

® D.T. Campbell and J.C. Stanley, Experimental and 
quasi-experimental designs for research. Boston: 
Houghton Mifflin Company, 1963. 
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to the evaluation by studying the 
effects of the reform on schools and 
student performance. 

The quantitative focus in evaluating 
Children Achieving was strongly 
supported by the following two-part 
rationale: First, the foundation for 
producing meaningful and relevant 
results is having a valid understanding 
of what is being studied. Researchers 
need to know what they are measuring 
and know that there is empirical 
support for their constructs. Second, 
once an empirical understanding of 
reform components is identified, the 
accuracy and generalization of results 
depends on thoughtful analysis to 
maximize the probability of accurate 
and comprehensive results. 

DATA ANALYSIS AND 
RESEARCH QUESTIONS 

The work of the quantitative data 
analysis team was guided by one 
overarching question: How has Children 
Achieving impacted the School District 
of Philadelphia's students, teachers, 
and schools? 

To explore the impact and effects of 
the Children Achieving reform, a multi- 
stage process of data analysis was 
conducted. First, we wanted to 
investigate whether the Children 
Achieving reforms demonstrated 
significant relationships with student 
achievement after controlling for key 
variables that might confound results. 
Second, we wanted to see if there were 
any significant influences on student 
success among the poorest schools 
(i.e., school environment and teacher 
demographic variables) in an effort to 



identify what helps these students 
succeed. 

A central concern was the degree of 
accuracy with which we could address 
the potential effect of the Children 
Achieving reform. Accurate 
measurement of its effects was a direct 
function of the degree to which we 
accurately measured the dimensions of 
the reform itself. In other words, before 
we were able to look at relationships 
between reform components and 
possible outcomes, we needed to be 
certain that we were working with valid 
constructs. While the teacher survey 
was developed with specific reform 
components drawn from the theory of 
action and the District's work plans, 
steps were taken to empirically 
document their existence in the 
schools. Consequently, our first 
research question addressed the 
construct validity of the teacher survey. 

Once meaningful constructs were 
established, data were analyzed using 
multiple methods and procedures. 
These methods are outlined in detail 
below. They represented a reliable way 
to address the proposed hypotheses 
about the effects of the reforms. 

Overall, the quantitative analysis of 
Children Achieving addressed four 
specific research questions: 

• What meaningful and measurable 
components composed the Children 
Achieving effort? 

• What was the relationship between 
Children Achieving and student 
achievement? 
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• Did identified school-level 
characteristics (i.e., school safety, 
school climate, instructional 
obstacles), teacher demographics, 
school conditions, and Children 
Achieving reform variables 
significantly relate to fourth grade 
reading achievement? 

• Did identified school-level 
characteristics, fourth grade reading 
achievement, and certain Children 
Achieving reform variables 
significantly relate to teacher- 
reported school conditions and 
other aspects of the Children 
Achieving reform? 

ELEMENTARY FOCUS 

Due to both the magnitude and the 
limitations of the data we collected, the 
results reported in this report pertain to 
a specific subgroup. Specifically, 
because elementary schools had the 
highest response rates to the teacher 
survey, the sample size for this group 
was most appropriate for the analyses. 
Because the fourth grade test scores 
are used in the District's accountability 
system, these students were 
highlighted for analyses. The test data 
came from the administration of the 
Stanford Achievement Test-?* Edition. 
Moreover, reading was selected as the 
focus subject area. Reading has been 
widely documented as a fundamental 
skill to be mastered in the elementary 
school years, and has also been shown 
to be related to future school 
performance. Analyses looking at fourth 
grade mathematics and science 
achievement, as well as analyses for 
middle school and high school grades 
are beyond the scope of the current 



report, but have been recognized as 
priority areas for future analyses. 

TEACHER RESPONDENTS 

Tables 1, 2, and 3 provide information 
about the Philadelphia School District 
teachers who responded to the 1998- 
1999 survey, and are cited from an 
earlier Consortium for Policy Research 
in Education (CPRE) report on teacher 
quality. For a more detailed description 
of the teachers who participated in the 
Children Achieving Challenge, please 
refer to Recruiting and Retaining 
Teachers: Keys to Improving the 
Philadelphia Public Schools.^ 

STUDENT POPULATION 

Student-level data aggregated at the 
school level was retained for those 
schools with teachers who returned the 
survey. In general, the demographic 
breakdown of the students represented 
in all analyses are representative of the 
School District of Philadelphia. 

TEACHER SURVEY 

CPRE administered the 1999 
Philadelphia teacher survey in June 
1999. The survey consisted of over 300 
items asking teachers' perception on 
school safety, professional community, 
school leadership, school conditions, 
professional development, components 
of the Children Achieving reform, 
teacher background, and classroom 
practice in mathematics, reading. 



’ S. Watson, Recruiting and retaining teachers: Keys 
to improving the Philadelphia public schools. 
Philadelphia: Consortium for Policy Research in 
Education, University of Pennsylvania, 2001. 
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English, or language arts. Data from the 
teacher survey were used to construct 
variables representing school 
conditions, the Children Achieving 
reform, and teacher qualifications. 
Please refer to Appendix A for a 



complete description of the 
development of the teacher survey, 
detailed description of variable criterion 
and categorization, and the procedures 
for distribution. 



TABLE 1. PROFILE OF TEACHERS IN THE SCHOOL DISTRICT OF PHILADELPHIA FROM 
COMBINED DATA SOURCES: 1998-1999 TEACHER SURVEY AND DISTRICT DATA 





Ethnicity 


Gender 


Qualifications 


Experience 


N 


Percent 

White 


Percent 

African 

American 


Percent 

Male 


Percent with 
Master's 
Degree and 
Higher 


Percent 
with More 
than 15 
Years Total 
Experience 



Elementary 


3,393 


67 


25 


13 


44 


55 


Middle 


889 


58 


37 


17 


43 


41 


High 


1,264 


69 


22 


52 


62 


70 


Survey Total 


5,731’ 


65 


26 


25 


49 


56 


District Total 


10,415 


63 


34 


26 


58 


44' 



' 185 teachers teach in schools with other grade configurations. 



^Percent of teachers with 15 or more years in the District. 



TABLE 2. HIGHEST LEVEL OF QUALIFICATION OF TEACHERS AND TOTAL WITH AT 
LEAST A MASTER'S DEGREE IN 1998-1999, BY SCHOOL LEVEL AND SUBJECT IN 
SCHOOL DISTRICT OF PHILADELPHIA (IN PERCENTS) 





Bachelor's 


Master's Only 


Master's and 
Higher 


Master's Total 


Grade Level 


Elementary 


27 


29 


44 


73 


Middle 


29 


29 


43 


72 


High 


17 


21 


62 


83 


Total 


25 


27 


49 


76 


Academic Subject 


Math 


24 


34 


25 


59 


Science 


33 


26 


25 


51 


Other Subjects 


34 


25 


40 


65 




1 






Source: 1998-1999 CPRE teacher survey. 
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TABLE 3. PERCENT OF TEACHERS: IN FIRST YEAR OF TEACHING, WITH FIVE OR FEWER 
YEARS OF TEACHING, OR MORE THAN 20 YEARS TEACHING IN CURRENT SCHOOL, 
AND IN TOTAL, 1 998-1 999, SCHOOL DISTRICT OF PHILADELPHIA (IN PERCENTS) 



First Year Teaching Five or Fewer Years Over 20 Years Teaching 

Teaching 





In this 
school 


In total 


In this 
school 


In total 


In this 
school 


In total 


Elementary 


17 


6 


48 


22 


12 


45 


Middle 


19 


7 


51 


29 


6 


28 


High 


15 


4 


44 


15 


17 


61 


Total Average 


16 


6 


48 


21 


13 


46 



Source: 1998-1999 CPRE teacher survey. 

Note: In an effort to ensure the integrity of the sample, only surveys that were returned with at least 40 percent 
completion were retained for analysis. Under these guidelines, 3,366 teachers from 133 schools were included in 
the sample, with the majority of the respondents being elementary school teachers. The response rates of teachers 
by grade level are shown in Table 4. 



TABLE 4. RESPONSE RATES AND TABLE 5. PROFILE OF SCHOOL DISTRICT 

NUMBERS TO THE CPRE OF PHILADELPHIA STUDENTS BY 

TEACHER SURVEY, 1998-1999 ETHNICITY, 1998-1999 (IN PERCENTS) 





Percent 


N 




White 


African 


Asian 


Latino 


Native 




Response 


Response 






American 


American 




American 


Elementary 


66 


3,254 


Elementary 


18 


64 


5 


13 


.2 


Middle 


54 


852 


Middle 


14 


69 


3 


14 


.1 


High 


58 


1,224 


High 


21 


62 


6 


10 


.3 


Total 


63 


5,330 


Total 


18 


65 


5 


12 


.2 



Note: The percentage breakdowns presented above are 
commensurate with the students used in the analyses for this report. 



FOURTH GRADE READING 
ACHIEVEMENT 

As part of the accountability system of 
the Children Achieving reform, the 
Stanford Achievement Test-9th Edition 
(SAT-9) in mathematics, reading, and 
science was administered annually to 
fourth, eighth, and eleventh grade 
students since 1996. CPRE collected 
school-level SAT-9 achievement scores 
in reading, mathematics, and science 



since 1996. As the school-level reading 
achievement results had the widest 
range in distribution and were highly 
correlated with both mathematics and 
science achievement, a decision was 
made to use reading achievement as 
the dependent variable. The average 
normal curve equivalent (NCE) scores 
from 1996 through 1999 for 173 
elementary schools (both traditional 
elementary and K-8 schools) were then 
adjusted to take into account the 
gradual expansion of the population of 
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students taking the test.'° The final 
dependent variable used was the 
adjusted school-level fourth grade 
reading achievement from 1996 to 
1999. 

DISTRICT-REPORTED 
SCHOOL-LEVEL 
DEMOGRAPHIC DATA 

The School District of Philadelphia 
collects records annually on each of its 
schools including records on school 
poverty, student mobility, student 
attendance, staff mobility, and total 
enrollment. CPRE used the 1999 
school-level demographics data in this 
analysis. 

EVALUATION AND 
RESEARCH 
QUESTIONS AND 
FINDINGS 

QUESTION 1 : WHAT 
MEANINGFUL AND 
MEASURABLE 

COMPONENTS COMPOSED 



For the HLM analysis, average normal curve 
equivalent scores for every school with a fourth 
grade were adjusted to account for the changing 
population of students who took the test each year. 
This primarily affected baseline scores (1996) when 
dramatically fewer students took the SAT-9 than in 
later years. Assuming that most of the students 
excluded from testing each year would have scored 
at lower levels on average than those who took the 
test, we adjusted each school's scores using the 
mean normal curve equivalent, its standard 
deviation, and the proportion of students who took 
the test each year. This process was undertaken in 
order to assure that we were comparing test scores 
from the same population of students annually. 



THE CHILDREN ACHIEVING 
EFFORT? 

RATIONALE AND HYPOTHESES 

The first critical step of the quantitative 
research team was to empirically 
document what the teacher survey was 
attempting to measure. While the 
Children Achieving theory of action 
provided the theoretical basis for 
survey construction, the survey's latent 
structure, or empirical inherent 
meaning, needed to be statistically 
investigated in order to confirm what 
specific constructs or factors the survey 
items were measuring. Specifically, 
principal-axis factor analysis was used 
to identify the inherent structure (i.e., 
combination of factors) captured by the 
survey items." Appendix B provides a 
detailed description of the empirical 
steps followed to arrive at the final 
factor solution used for all subsequent 
analyses. In short, a series of 
exploratory and confirmatory 
procedures were executed to identify 
the set of factors that most reliably and 
meaningfully represented the latent 
structure of the survey items. 

In this analysis, a factor refers to a 
group of items that, through statistical 
analysis, hold together. Based on the 
composition of the item (i.e., what the 
item says), a name for the factor is 
given to best describe the overall 
theme or meaning of the item group. 



” Squared multiple correlations were used as the 
initial communality estimates for the common factor 
analyses. Additionally, promax solutions were run at 
varying levels of power, k=3, 5, 7, and each oblique 
solution was compared to the final orthogonal 
solution to determine the most parsimonious 
explanation. 
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One of the nnost innportant uses of 
factors developed from the survey is 
that they can then be used as variables 
in further analysis. Here, a variable is a 
general term that refers to a quantified 
concept that is part of an empirical 
question being tested. 

It is important to note that in addition 
to the empirical steps taken to ascertain 
and describe the final factor solution, 
members of the qualitative research 
team were also consulted to describe 
the different constructs based on the 
rubrics they used to describe schools in 
their work. Through collaborating with 
the qualitative research team, 
observations and hypotheses were 
examined in light of the empirical 
findings to conceptually frame and 
define the final factor solution. 

RESULTS 

Factor analysis of the teacher survey 
revealed eight independent factors. 
Each of the eight factors fell into one of 
two groups. The first group. Children 
Achieving Reform Components, 



consisted of four factors: Small Learning 
Communities; the Teaching and 
Learning Network; Curriculum, 
Assessment, and Accountability; and 
Teacher Practice. The second group of 
factors. School Characteristics, included 
the following: School Climate, School 
Safety, Obstacles to Student Learning, 
and Teacher Professional Community. 
Each of the eight factors demonstrated 
high internal consistency. Here, 
Cronbach Alpha coefficients reflect the 
degree to which the items in a factor 
relate to each other. Table 6 shows the 
survey factors and their internal 
reliability coefficients. 

The first group of factors relate to 
different components of the Children 
Achieving reform, and more 
particularly, reflect the degree of 
implementation of different aspects of 
Children Achieving. Small Learning 
Communities consisted of sub-units of 
schools organized to improve the 
conditions of teaching and learning and 
strengthen relations between teachers 



TABLE 6. TEACHER SURVEY FACTORS AND CRONBACH'S ALPHA COEFFICIENTS 



Factor Name 


Cronbach's Alpha 


Children Achieving Reform Components 


Small Learning Communities 


.91 


Teaching and Learning Networks 


.93 


Curriculum, Assessment, and Accountability 


.89 


Teacher Practice 


.95 


School Characteristics 


School Climate 


.79 


School Safety 


.84 


Obstacles to Student Learning 


.89 


Teacher Professional Community 


.94 
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and students. Sample items of survey 
questions that composed this factor 
include: "My small learning community 
has a clear vision that shapes my 
curriculum and instruction," "The 
teachers in my small learning 
community make decisions about 
curriculum and instruction," and 
"Identified individual intervention 
strategies for students who needed 
additional assistance." 

Teaching and Learning Network (TLN) 
is a second Children Achieving reform 
component. It was originally designed 
to be the professional development 
arm of the District, and is characterized 
by items such as, "TLN staff have 
helped me identify and observe good 
teaching practices," "the TLN has given 
me help in dealing with students with 
special needs," and "the TLN staff in 
my cluster have the knowledge and 
skills they need to help me improve my 
classroom practice." 

The Curriculum, Assessment, and 
Accountability factor captured 
curricular resources offered as part of 
Children Achieving. This factor included 
items such as, "I have made changes in 
my teaching strategies to help our 
school achieve the year's performance 
index target," and "The Philadelphia 
Curriculum, Assessment, and 
Accountability Frameworks have led me 
to make changes in what I teach." 

Finally, Teacher Practice refers to 
different learning strategies 
implemented by teachers in the 
classroom, and is composed of items 
including, "Estimate the change this 
year in how often you asked students to 
work in pairs or small groups, critique 



other students, and write about 
something they read." 

The second group of factors represents 
different School Conditions or 
Characteristics. The first factor. School 
Climate, refers to different aspects of 
the school environment, such as 
student characteristics and student- 
teacher relations that are hypothesized 
to contribute to the learning 
environment. The instructional climate 
factor is described by items such as, 
"Students are generally well behaved in 
the classroom," and "To what extent 
do you feel respected by students at 
this school, the parents of your 
students, and central office staff." 

School Safety is characterized by items 
such as, "Applying disciplinary rules 
more consistently within the school," 
"Insisting that all students wear picture 
identification badges," and "Increasing 
the number of security personnel." 

Obstacles to Student Learning describe 
those things that impede student 
learning, and include items such as, 
"Inability to access community 
support," "Poor student attendance," 
and "High student mobility in and out 
of the school." 

Finally, Teacher Professional 
Community captures those items that 
describe the dynamics among teachers, 
teacher-principal relations, teacher 
collaboration, and principal leadership. 
The Teacher Professional Community 
factor is characterized by items such as, 
"Teachers support the principal in 
enforcing school rules," "The principal 
takes a personal interest in the 
professional development of the 
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teachers," and "Teachers respect 
teachers who are trying new 
instructional approaches." 

CROSS-VALIDATION AND 
CONFIRMATORY ANALYSIS 

The proposed solution was then 
subjected to cross-validation and 
confirmatory factor analysis procedures 
to further substantiate its integrity. 
Cross-validation procedures involved 
the use of the Wrigley-Neuhause 
coefficient. In this procedure, the large 
sample was randomly bifurcated into 
two smaller groups, and then the 
smaller groups were compared to the 
larger one. Every possible combination 
of factors was compared, and 
coefficients were generated that 
indicated the extent of similarity across 
hypothesized like factors, as well as the 
extent of dissimilarity across 
hypothesized unlike factors.'^ Results of 
these analyses revealed acceptance 
coefficients of congruence. 

Finally, to confirm the make-up of the 
final eight-factor solution, the items 
retained during exploratory analysis 
were subjected to a confirmatory 
multiple-group cluster analysis.'^ 
Hypothesized cluster membership was 
based on the exploratory analyses, and 
items were permitted to migrate 
iteratively to clusters that better 
explained item variance. In this stage of 
the analysis, the retention of items in 
the original hypothesized groupings 
serves to confirm the integrity of the 

E. Guadagnoli and W. Velicer, "A comparison of 
pattern matching indices." Multivariate Behavioral 
Research 26 (1 991). pp. 323-343. 

H. Harman, Modem factor analysis (third edition). 
Chicago: University of Chicago Press, 1976. 



exploratory structure. In all cases, the 
empirically-derived structure (i.e., eight- 
factor solution) was found to be 
superior to any of the alternatives, 
explaining a higher proportion of the 
item variance than the alternatives. 

QUESTION 2: DID THE 
CHILDREN ACHIEVING 
REFORM SIGNIFICANTLY 
RELATE TO THE INITIAL 
STATUS AND GROWTH OF 
STUDENT ACHIEVEMENT? 

RATIONALE AND HYPOTHESES 

The second stage of analysis consisted 
of looking at the possible relationships 
between factors from the teacher 
survey, relevant demographic variables, 
and student achievement. Specifically, 
HLM, or hierarchical linear modeling 
analysis, was used to ascertain which 
combination of variables most 
significantly related to students' initial 
achievement status levels by school, as 
well as school-level student 
achievement growth over four years 
(1996-1999). It should be noted that at 
the time of the current analyses, 
student achievement data for the 1999- 
2000 school year was not yet available. 
HLM has been identified as an effective 
model of multi-level data structures.''* 

Poverty is one of the most important 
indices which has been consistently 
associated with low achievement. 
Specifically, poverty and other 
disproportionately high familial and 



" A.S. Bryk and S. Raudenbush, Hierarchical linear 
models: App//cat/ons and data ana/ys/s methods. 
Newbury Park, CA: Sage Publications, 1992. 
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community stressors common to urban 
areas are hypothesized to threaten 
children's healthy development and 
academic adjustment.’^ Evidence clearly 
indicates that students with the lowest 
levels of achievement are 
disproportionately represented in high- 
poverty schools. 

Moreover, School Safety and Obstacles 
to Student Learning are two elements 
of school climate that have also shown 
to be related to students' school 
performance. For example, poor 
student attendance, lack of parent 
involvement,’^ and perceived sense of 
safety have been identified as 
impediments to student learning and 
achievement. Moreover, there is a 
significant amount of support in the 
literature stating that the nature of the 
professional learning community is also 
related to how students perform in 
school. Finally, Distributed Leadership 
and teacher satisfaction have also been 
generally identified in the literature as 
important agents in promoting 
students' achievement. 

A growing body of research positively 
links strong professional community to 
student achievement.’® Fullan calls for a 



" National Center for Education Statistics, Urban 
schools: The challenge of location and poverty. 

National Center for Education Statistics, Urban 
schools: The challenge of location and poverty. 
Garibaldi, "Four decades of progress and decline." 

" J.S. Eccles and R.D. Harold, "Parent-school 
involvement during the early adolescent years." 
Teachers College Record 94 (1994), pp. 555-587. 

M.G. Fullan and A. Hargreaves, Eds., 
Understanding teacher development. New York: 
Teachers College Press, 1992; K.S. Newman, "Local 
caring: Social capital and social responsibility in New 
York's minority neighborhoods." In A.S. Rossi, Ed., 
Caring and doing for others: Social responsibility in 



"reculturing" of schools that would 
produce collegial relationships 
characterized by high expectations for 
everyone's — adults and students — 
learning and performance. In such a 
transformation, staff would routinely 
reflect about their classroom practices, 
they would seek out new and promising 
ideas, and enlist the support of 
knowledgeable outsiders. Children 
Achieving sought to strengthen 
professional community by emphasizing 
results, raising expectations, and 
decentralizing decision-making. 
Performance targets would focus on 
school staffs' attention on outcomes 
and unify teachers in the pursuit of clear 
goals. District leaders would be 
persistent in their message that, "All 
children can learn if we believe they 
can." The expertise of teachers would 
be honored as faculties maintained 
authority over decisions about 
curriculum and instruction. Local school 
councils and small learning communities 
would offer ways for teachers to assess 
instruction, reflect, and plan. 

Decisions about which variables to 
include in our regression analysis 
stemmed from a specific conceptual 
model, which framed our 
understanding of how certain factors 
related to students' performance. This 
conceptual model, or theoretical 
rationale, came from the work of our 
qualitative research team, as well as 
from the theoretical literature. 



the domains of family, work, and community (pp. 

1 57-177). Chicago: University of Chicago Press, 

2001. A. Brodsky, PJ. O'Campo, and R.E. Aronson, 
"PSOC In community context: Multi-level correlates 
of a measure of psychological sense of community in 
low-income, urban neighborhoods." Journal of 
Community Psychology 27 (1999), pp. 659-679. 
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It was important to carefully consider 
what variables to include in HLM 
analyses. In Philadelphia, for example, 
as in many urban school districts, many 
student, school, and teacher 
characteristics are correlated. The 
District's own research demonstrated 
that experienced teachers transferred 
to schools with higher achievement 
levels, lower levels of poverty, and 
fewer minority students. The fact that 
many of our variables were highly 
correlated was important. If all variables 
were included in equation models, it 
was likely that they would cancel each 
other out. In addition, because we had 
so many variables, including them all in 
the analyses would put a strain on the 
assessment, and make interpretation of 
results very complex. 

After extensive preliminary analyses, 
and after examining descriptive 
statistics and correlation information, 
variables that emerged as the most 
distinct from others were included in 
the analyses. 

We hypothesized that student 
achievement is a function of effective 
instructional practice, and that basic 
school conditions and the quality of the 
school environment are two important 
precursors to quality instruction. Based 
on this theoretical rationale, the 
variables that were included in the HLM 
analysis included poverty, and the 
factors from the teacher survey (School 
Conditions, Teacher Professional 
Community, and Reformed Practice). 



EQUATION MODELS FOR HLM 
ANALYSIS 

Various statistical methods were used 
to thoroughly investigate the 
relationship between the Children 
Achieving reform and longitudinal 
changes in reading achievement of 
fourth grade students. In particular, we 
used repeated measures analysis of 
variance to understand initially if there 
was a significant difference between 
the scores over time. When that 
analysis showed that there were indeed 
significant differences in school 
achievement overtime F(1, 172) = 
135.93, p < .001, we employed 
hierarchical linear modeling to 
understand what school-level variables 
predict the observed changes over 
time. 

Graphs of the schools' achievement 
scores over time were plotted to see 
what trends emerged. These graphs of 
individual school average achievement 
overtime revealed that there are 
different growth curves for different 
schools. For instance, some schools 
have a clear linear growth curve, with 
their lowest achievement in 1996, and 
their highest achievement in 1999. 
Others, however, show more complex 
patterns, including low achievement in 
1996, climbing to a high level in 1998, 
and then decreasing slightly in 1999. 
After considering parabolic models, we 
concluded that the best predictive 
model to capture growth was the linear 
growth curve model, a simple model 
that assumes a linear relationship 
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between time and achievement.'’ This 
was done by using HLM. 

HLM was developed specifically to 
address the hierarchical levels found in 
education, such as the organization of 
districts of schools, classrooms, and 
students. The following HLM analysis 
addressed another type of data 
hierarchy: student achievement 
measured over time. As HLM allows for 
the partitioning of variance of the 
dependent measure by levels, we can 
use this method to demonstrate which 
school variables contributed to schools' 
initial achievement scores (i.e., 1996) 
and which contributed to growth (i.e., 
change from 1996 to 1999). This 
partitioning is achieved by formally 
modeling independent variables from 
each level. Below is the initial HLM 
model used in the analysis: 

Initial HLM Model: 

Level-1 Model 
Y = B0+ B1*(Time) + R 

Here, Y is the average fourth grade 
reading achievement score, BO is the 
1996 score or the initial status 
parameter, B1 is the linear growth 
parameter or the growth rate, and R is 
the error. The Level-1 model shows that 

19 

It should be noted that the variability of school 
scores could be a potential problem when 
conducting analyses (i.e., school average scores can 
be variable from year to year). In this case, the 1 996 
scores were the baseline year for the SAT-9 
administration in the District. Consequently, while it 
has been noted as something important to consider, 
it would be difficult to specifically ascertain score 
variability in this situation. School-level scores were 
utilized in analyses because student-level scores, 
which have been found to be less variable, were not 
available to us at this time. 

Bryk and Raudenbush, Hierarchical linear models. 



we are modeling student achievement 
as a function of its intercept and time. 

Level-2 Model 
B0= GOO + UO 
B1= G10 + U1 

Note that each of the parameters from 
the Level-1 model (BO and B1) are 
modeled as the dependent variable in 
the Level-2 model. Both BO and B1 are 
allowed to randomly vary with their own 
intercept, GOO and G10, respectively. 

Our final HLM model appears below: 

Level-1 Model 
Y= BO + B1*(TIME) + R 

Level-2 Model 

BO = GOO + G01*(LOWINC99) + UO 
B1 = G10 + G11*(TSAFE) + 
G12*(TOBST) + G13*(TLEAD) + 
G14*(TTLN) + G15 + U1 

Here, BO is the initial status that is 
being predicted by its intercept (GOO), 
incremental effect of poverty (G01), and 
error (UO). B1 is the linear growth 
parameter, or growth rate, that is being 
predicted by its intercept (G10), 
incremental effect of poverty 
concentration (G11), School Safety 
(G12), incremental effect of Obstacles 
to Student Learning (G13), incremental 
effect of Teacher Professional 
Community (G14), incremental effect of 
the Teaching and Learning Network 
(G15), and error (U1). 

We have developed a model that says 
that 1996 reading achievement (BO) can 
be predicted by poverty; while the 
growth rate from 1996 to 1999 (B1) can 
be predicted by School Safety, 
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Obstacles to Student Learning, Teacher 
Professional Community, and the 
Teaching and Learning Network. 

RESULTS 

Examining the growth in schools' 
reading scores using HLM revealed 
three important findings. First, it 
showed that low income was 
significantly related to the initial status 
of school achievement (i.e., the poorer 
the school, the lower the initial status of 
children's mean achievement in that 
school). This finding is supported by 
qualitative research reporting that 
schools with significantly high 
concentrations of poverty had higher 
percentages of students scoring Below 
Basic levels at the start of the reform 
than other schools at their level. 

Second, our HLM analysis revealed that 
poverty is also associated with growth 
in achievement scores during the 
Children Achieving reform. Specifically, 
poorer schools experienced 
significantly more growth in the 
percentages of students scoring above 
the Basic Level over the course of the 
reform. Our qualitative research 
supports this finding by documenting 
cases in which some of the more 
impoverished schools experienced 
more growth in achievement scores 
than the average amount of growth for 
their school level. 

The third important finding from the 
HLM analysis is that Teacher 
Professional Community was 
significantly related to the rate of 
growth for children's achievement 



scores (i.e., the stronger Teacher 
Professional Community reported by 
the teachers to be in a particular 
school, the faster the rate of growth in 
children's achievement scores in that 
school). This also appears to be 
strongly reflected in the qualitative 
accounts of factors that affect student 
achievement growth. 

Results from the initial HLM model (see 
Table 7) show that the average starting 
point for all schools in 1996 was 34.49 
NCE scores. On average, all schools 
improved their results in reading at a 
rate of 2.80 NCE scores per year for the 
four years measured. 

Table 8 displays between-school 
differences on the average starting 
point and average rate of student 
achievement growth. Because a 
statistically significant difference 
between schools in their starting point 
was found (68.09, p < .01), this 
suggests that variability resulting from 
this difference should be taken into 
account in the analysis. 

Moreover, our results indicate that 
schools' growth patterns, even though 
they were different, as noted earlier, 
were not significantly different over 
time (0.36, p > .05). This finding 
suggests that it is important to examine 
school conditions and variables as 
predictors of growth in achievement 
over time. The data in Tables 7 and 8 
give credence to our analysis of school 
conditions and variables as predictors 
of growth rates. 
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TABLE 7. ESTIMATION OF THE AVERAGE 1996 SCORE 
AND GROWTH RATE FROM 1996 TO 1999 



Parameter 


Coefficient 


Standard Error 


1 996 Average Score 


34.49** 


(0.69) 


Growth Rate from 1996-1999 


2.80** 


(0.16) 



Note:**p < .01. 



TABLE 8. BETWEEN-SCHOOL DIFFERENCES ON THE 
AVERAGE STARTING POINT AND RATE OF INCREASE 



Variance Component 
1996 Average Score 
Growth Rate from 1996-1999 



Coefficient 

68.09** 

0.36** 



Note: ** p < .01. 

TABLE 9. PREDICTING INITIAL SCORE BY POVERTY 



Parameter 


Coefficient 


Standard Error 


Intercept 


40.63** 


(0.41) 


Poverty 


-0.43** 


(0.03) 



Note. ** p < .001. 



Our initial findings showed a negative 
correlation between a school's initial 
status and the rate of growth (r = -.47). 
This means that schools with lower 
scores in 1996 had higher growth rates 
than schools with higher initial scores. 

The next stage of our analysis involved 
investigating how school-level variables 
were related to each school's initial 
status and to the growth of student 
achievement. Results from the final 
HLM model revealed that poverty was a 
significant predictor of initial 
achievement status (see Table 9). The 
negative coefficient (-0.43, p < .01) 
means that the poorer the school, the 
lower its achievement in 1996. This is 



supported by qualitative reports that 
schools with significantly high 
concentration of poverty had higher 
percentages of students scoring Below 
Basic levels at the start of the reform 
when compared to other schools at 
their level. 

Examining results from the final HLM 
model that looked at how school 
conditions and school variables affected 
growth in achievement, we found that 
Teacher Professional Community was a 
significant predictor of growth over 
time, controlling for incremental effects 
of School Safety, Obstacles to Student 
Learning, and the implementation of 
the Teaching and Learning Network 
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TABLE 10. PREDICTING GROWTH BY SCHOOL SAFETY, 
INSTRUCTIONAL OBSTACLES, STAFF COLLEGIALITY, AND 
IMPLEMENTATION OF THE TEACHING AND LEARNING NETWORK 



Parameter 


Coefficient 


Standard Error 


Intercept 


-9.84* 


(3.94) 


School Safety 


-0.02 


(0.06) 


Obstacles to Student Learning 


0.03 


(0.05) 


Teacher Professional Community 


0.17** 


(0.05) 


Teaching and Learning Network 


0.05 


(0.05) 



Note: * p < .05, ** p < .01. 



(see Table 10 ). That is, the more 
distributed the leadership, the faster 
the rate of growth in achievement 
scores. This appears to be strongly 
supported by the qualitative accounts 
of factors that affect student 
achievement growth. Other school 
conditions such as School Safety had a 
non-significant negative relationship 
with the growth rate of student 
achievement, indicating the less safe a 
school, the lower the rate of growth. 

On the other hand, the fewer Obstacles 
to Student Learning, the higher the 
growth rate. And lastly, our measure of 
the Children Achieving reform had a 
non-significant positive relationship 
with the growth rate. 

SUMMARY 

Our longitudinal analysis of fourth 
grade reading achievement scores 
reveals that elementary achievement 
scores grew an average of 2.8 NCE 
points per year. Across the four years of 
reform, this adds up to an average total 
growth of 8.4 NCE points, about 2/5ths 
of a standard deviation. According to 



Slavin and Fashola,^^ this amount of 
growth is educationally significant. 

Schools that have a higher 
concentration of poor students tended 
to have the lowest 1996 average 
scores. However, these schools also 
grew the fastest over time. 

After controlling for the incremental 
effects of basic school environmental 
factors such as School Safety and 
Obstacles to Student Learning, Teacher 
Professional Community remained a 
significant predictor of growth in 
achievement scores over time. 

Finally, Children Achieving reform 
variables (i.e.. Curriculum, Small 
Learning Communities, Teaching and 
Learning Communities, and Reformed 
Practice) did not emerge as statistically 
significant in predicting the rate of 
growth in school achievement. One of 
the limitations of the HLM procedure is 
its ability to control for variables that 



R.E. Slavin and O.S. Fashola, Show me the 
evidence! Proven and promising programs for 
America's schools. Thousand Oaks, CA; Conwin 
Press, 1998. 
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might be correlated. Based on the HLM 
analyses, results seem to suggest that 
certain variables were masked by 
others, supporting our earlier 
hypothesis that many of our school 
characteristics and Children Achieving 
reform variables are related. More 
specifically, when models that included 
isolated groupings of school 
characteristics and reform variables 
were examined, different variables did 
demonstrate some significant 
relationships with the initial student 
achievement status and growth of 
student scores. In reality, however, the 
reform component and school 
characteristics exist simultaneously, and 
it was critical to reflect this in the 
equation models. When all variables 
were entered simultaneously, far fewer 
significant relationships emerged, 
suggesting that the effects of certain 
components were being cancelled out 
by the presence of other reform 
variables also included in the model 
that were related to each other (multi- 
collinearity). 

Given these circumstances, the 
quantitative analysis team conducted 
further analyses using alternative 
methods that better addressed the 
issue that many of the variables were in 
fact related. This was done using 
another analysis method that better 
controls for all variables in a given 
equation model: logistic regression. 



QUESTION 3: DO 
IDENTIFIED SCHOOL-LEVEL 
CHARACTERISTICS (I.E., 
SCHOOL SAFETY, SCHOOL 
CLIMATE, OBSTACLES TO 
STUDENT LEARNING), 
TEACHER DEMOGRAPHICS, 
SCHOOL CONDITIONS, 

AND CHILDREN ACHIEVING 
REFORM VARIABLES 
SIGNIFICANTLY RELATE TO 
FOURTH GRADE READING 
ACHIEVEMENT? 

RATIONALE AND HYPOTHESES 

In an effort to more specifically 
understand and unpack how each of 
the school-level characteristics and 
Children Achieving reform components 
related to student achievement, logistic 
regression analysis was used. Logistic 
regression allowed us to look at the 
relative relationship of each of our 
independent variables (i.e., school 
characteristics and Children Achieving 
reform variables) with student 
achievement controlling for all other 
independent variables in the analysis. 
Specifically, logistic regression is a 
regression procedure used when the 
outcome or dependent variable is 
categorized into two groups (i.e., 
average or above the mean reading 
scores versus below the mean reading 
scores). In essence, logistic regression 
allows you to calculate the probability 
of ending up in one of the two outcome 
groups (i.e., below the mean in reading) 
based on whether or not you have 
certain independent variable 
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characteristics (i.e., in a school with low 
Teacher Professional Community). In 
this procedure, the odds ratio is a 
practical statistic that provides an index 
to interpret the relative likelihood of 
being classified in a specific outcome 
group based on having specific 
characteristics or not. 

For this set of analyses, the relative 
import of teacher demographics, school 
characteristics, and Children Achieving 
reforms on fourth grade reading 
achievement was investigated. Table 1 1 
displays the variables included in the 
model. 



RESULTS 

Looking at the entire fourth grade 
sample, less teacher-reported School 
Safety and more teacher-reported 
Obstacles to Student Learning 
significantly increased the odds of 
children scoring below the mean in 
reading achievement. In other words, 
schools with less teacher-reported 
safety were 1 .45 times more likely to 
have a mean fourth grade reading 
achievement score below the mean. 



TABLE 11. EXPLANATORY AND OUTCOME VARIABLES FOR LOGISTIC REGRESSION 
MODELS INVESTIGATING RELATIONSHIPS TO STUDENT ACHIEVEMENT 



Independent (Explanatory) Variable 


Dependent (Outcome) 
Variable 


Teacher Demographics 


Fourth Grade Reading 


Amount of Education 
Certification Status 

Number of Years Teaching in the District 
School Characteristics 
Poverty 

School Size (Student Enrollment) 

Survey-measured School Characteristics 
School Safety 

School Obstacles to Student Learning 

Survey-measured Children Achieving Reform Components 
Teacher Professional Community 
Small Learning Communities 


Achievement 



TABLE 12. ODDS RATIOS FOR RELATIONSHIPS BETWEEN STUDENT, 
TEACHER CHARACTERISTICS, CHILDREN ACHIEVING REFORMS, AND 
FOURTH GRADE READING ACHIEVEMENT: TOTAL FOURTH GRADE SAMPLE 



Explanatory Variable 


Below the Mean Fourth Grade 




Reading Achievement 




Sig. Level 


Odds Ratio 


School Safety (Teacher-reported = Low) 


.0511 


1.45 


Obstacles to Student Learning (Teacher-reported = High) 


.0401 


1.33 
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TABLE 13. ODDS RATIOS FOR RELATIONSHIPS BETWEEN STUDENT, 
TEACHER CHARACTERISTICS, CHILDREN ACHIEVING REFORMS, AND FOURTH GRADE 
READING ACHIEVEMENT: EXTREME POVERTY SAMPLE 



Explanatory Variable 


Below the Mean Fourth Grade 
Reading Achievement 




Sig. Level 


Odds Ratio 


Low Teacher- re ported School Safety 


.0001 


1.57 


High Teacher-reported Obstacles to Student Learning 


.0500 


1.41 



When examining the extreme poverty 
sample, the same relationships are not 
only observed between school safety, 
instructional obstacles, and fourth 
grade reading achievement, but they 
emerge as more significant. In this 
group, children in schools where 
teachers report concerns about safety 
are 1 .57 times more likely to score 
below the mean in reading 
achievement. Moreover, these same 
children are now 1.41 times more likely 
to demonstrate poor reading 
achievement if they are in schools with 
more teacher-reported obstacles to 
student learning. 

QUESTION 4: DO 
IDENTIFIED SCHOOL-LEVEL 
CHARACTERISTICS, 

FOURTH GRADE READING 
ACHIEVEMENT, AND 
CERTAIN CHILDREN 
ACHIEVING REFORM 
VARIABLES SIGNIFICANTLY 
RELATE TO TEACHER- 
REPORTED SCHOOL 
CONDITIONS AND OTHER 



ASPECTS OF CHILDREN 
ACHIEVING? 

RATIONALE AND HYPOTHESES 

In the second set of logistic regression 
analyses, we looked at the relative 
import of Teacher Demographics, 
School Characteristics, and Small 
Learning Communities on School 
Safety, School Obstacles to Student 
Learning, and Teacher Professional 
Community. Two sets of logistic models 
were run. The first set included the 
entire fourth grade sample, and the 
second set included those schools that 
were labeled as "extreme poverty." 
Extreme poverty was defined as those 
schools where at least 85 percent of 
their students qualified for free or 
reduced lunch. Please refer to 
Appendix C for a more detailed 
description of how this variable was 
created. Because poverty has been so 
widely recognized as a powerful 
deterrent to student achievement, it 
was important to our team to 
investigate potential effects of Children 
Achieving on different poverty groups. 
In other words, while the logistic 
regression procedure examined 
relationships between reform 
components and student achievement 
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having statistically controlled for any 
potential effects of poverty, it was also 
important to study whether relationship 
patterns between school components 
and student achievement differed 
among different poverty sub- 
populations. For both samples three 
logistic models were run in which all of 
the independent models were used: 
one looking at unsafe schools, one 
looking at Obstacles to Student 
Learning, and one looking at less 
Teacher Professional Community. 

RESULTS 

Looking at the entire fourth grade 
sample, higher poverty, larger school 
size, and dissatisfaction with Small 



Learning Communities all significantly 
increased the odds of being in an 
unsafe school, a school with more 
Obstacles to Student Learning, and 
schools with a weaker sense of Teacher 
Professional Community. Finally, 
teachers with 6-12 years of experience 
were more likely than new teachers to 
be in a school with more Teacher 
Professional Community. Teachers with 
more than 10 years of experience in the 
District were more likely to be in safer 
schools, schools with fewer Obstacles 
to Student Learning, and schools with 
more Teacher Professional 
Development than first-year teachers. 



TABLE 14. EXPLANATORY AND OUTCOME VARIABLES FOR LOGISTIC 
REGRESSION MODELS INVESTIGATING RELATIONSHIPS TO 
CHILDREN ACHIEVING REFORM COMPONENTS 



Independent (Explanatory) Variable 


Dependent (Outcome) Variable 


Teacher Demographics 
Amount of Education 
Certification Status 

Number of Years Teaching in the District 


Being in an Unsafe School 

Being in a School with Obstacles to Student 

Learning 

Being in a School with Low Teacher Professional 
Community 


School Characteristics 
Poverty 

Fourth Grade Reading Achievement Scores 
School Size (Student Enrollment) 




Survey-measured Children Achieving 
Reform Components 
Small Learning Communities 
Curriculum, Assessment, Accountability 
Teaching Learning Network 
Reformed Practice 
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TABLE 15. ODDS-RATIOS FOR RELATIONSHIPS BETWEEN 
STUDENTS/TEACHERS CHARACTERISTICS AND CHILDREN ACHIEVING 
REFORMS: TOTAL FOURTH GRADE SAMPLE 



Explanatory Variable 



Safety Obstacles Collaboration 





Sig. 


Odds- 


Sig. 


Odds- 


Sig. 


Odds- 




Level 


Ratio 


Level 


Ratio 


Level 


Ratio 


Low Income (1999) 


.0000 + 


20.45 


.0000 + 


18.64 


.0007 - 


.23 


Student Enrollment (1999) 


.0034 + 


1.00 


.0001 + 


1.00 


.0103 - 


1.00 


Small Learning Communities 


.0000 + 




.0000 - 


.98 


.0000 + 


1.11 


Fourth Grade Reading 

Achievement 

Teacher Education 

Bachelor's 

Master's 

Master's + 


.0360 - 


.69 






.0087 + 


1.02 


Teacher Teaching Experience 
r year 
2-5 years 
6-1 0 years 










.0334 + 


1.39 


11-15 years 
>15 years 


.0290 - 


.74 


.0316 + 


1.53 


.0418 + 


1.36 



These findings suggest relationships 
between certain school characteristics 
and the degree to which certain 
aspects of the Children Achieving 
reform were operating in a school, as 
perceived by the teachers. Specifically, 
teachers with more experience were 
more likely to be in schools with fewer 
Obstacles to Student Learning, and a 
greater sense of Teacher Professional 
Community. Schools with higher fourth 
grade reading scores were more likely 
to have a greater sense of Teacher 
Professional Community. Schools with 
Small Learning Communities were more 
likely to have fewer Obstacles to 
Student Learning and more Teacher 
Professional Community. 

Having previously documented through 
the HLM analysis that Teacher 
Professional Community was found to 
relate to the growth of reading 
achievement over time, it is important 



to note that our analysis shows that 
certain school characteristics are 
indirectly related to student 
achievement. Teacher experience and 
Small Learning Communities have an 
indirect influence on fourth grade 
reading achievement in the presence of 
a better-developed Teacher 
Professional Community. 

Looking at the extreme poverty sample, 
smaller schools were more likely to 
have fewer Obstacles to Student 
Learning and a greater sense of 
Teacher Professional Community. 
Schools with higher fourth grade 
reading scores were more likely to have 
a greater sense of Teacher Professional 
Community. Moreover, teachers 
engaging in Small Learning 
Communities were more likely to be in 
safer schools, have fewer Obstacles to 
Student Learning, and experience a 
greater sense of Teacher Professional 
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TABLE 16. ODDS-RATIOS FOR RELATIONSHIPS BETWEEN STUDENTS/TEACHERS 
CHARACTERISTICS AND CHILDREN ACHIEVING REFORMS: EXTREME POVERTY SAMPLE 



Explanatory Variable 


Safety 


Obstacles 


Collaboration 




Sig. Odds- 


Sig. 


Odds- 


Sig. 


Odds- 




Level Ratio 


Level 


Ratio 


Level 


Ratio 


Student Enrollment (1999) 




.0225 H - 


1.00 


.0212 - 


1.00 


Small Learning Communities 


. 0000 + 1.02 


.0000 - 


.98 


.0000 H - 


1.11 


Fourth Grade Reading Achievement 








.0087 H - 


1.02 



Teacher Education 
Bachelor's 

Master's .0406 - .77 

Master's + 

Teacher Teaching Experience 

r'year 

2-5 years 

6-10 years 

11-15 years 

>15 years 



Community. Finally, teachers with more 
than Master's Degrees were more likely 
to be in schools with fewer Obstacles to 
Student Learning than teachers with 
Bachelor's degrees. 

Tables 15 and 16 show the significance 
and odds-ratios of the variables that 
relate to fourth grade reading 
achievement scores. It is important to 
note that while many of the 
relationships in the total and high- 
poverty concentration samples are 
similar, the odds-ratios in the high- 
poverty concentration sample are 
larger, indicating a higher likelihood of 
the specific outcome. 

To summarize, well-implemented Small 
Learning Communities appear to 
provide good learning environments for 
students regardless of their poverty 
level. Moreover, Small Learning 
Communities are better implemented 
in safer schools, schools in which there 
is more Teacher Professional 



Development, and schools with fewer 
Obstacles to Student Learning. 

SUMMARY OF 
FINDINGS 

The results of our analyses provide 
support for the hypothesis that 
Children Achieving reforms are related 
to students' school achievement, and 
that Children Achieving reform 
components demonstrate significant 
relationships to fourth grade reading 
achievement. Specifically, our general 
conclusions are a function of a 
sequential set of analysis steps taken to 
pursue our hypotheses. 

First, we analyzed the teacher survey to 
understand what teacher, school, and 
reform constructs the survey was 
measuring. The following constructs 
were found: 

• Basic School Environment (Quality 
of School Environment, School 
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Safety, School Obstacles to Student 
Learning) 

• Teacher Professional Community 
(Teacher Professional Community, 
Teacher Leadership) 

• Reform and Practice (Small Learning 
Communities, Teaching and 
Learning Network, Curriculum, 
Assessment, and Accountability) 

Next, we used these constructs and 
other relevant variables (i.e., low 
income) in an HLM analysis to see 
whether these reforms had significant 
effects on the growth of student 
achievement. We found in our 
hierarchical model that poverty 
explained the initial achievement status 
of schools, and poverty and Teacher 
Professional Community explained 
growth in student achievement during 
the Children Achieving reform. 

Then we explored how certain school- 
level characteristics related to positive 
learning environments for students in 
the poorest schools, and how they 
related to the implementation of 
Children Achieving. We also 
investigated whether these 
relationships differed as a function of 
poverty concentration in an effort to 
ease out the impact of the reform 
components controlling for the 
powerful effects of poverty. Here we 
used logistic regression analysis, a 
procedure that better allowed us to 
assess relationships between sets of 
correlated variables and specific 
outcomes, using only the 1999 data. 
While these models are not designed to 
explain growth as HLM analysis is, they 
can help us understand how individual 



variables (while controlling for the 
effects of all other variables included in 
the model) affect the odds of schools or 
students being classified into a 
particular group (i.e., unsafe school, 
below the mean in reading 
achievement). 

The results of the logistic regression 
analyses indicate that schools with 
more Obstacles to Student Learning 
and schools with more safety concerns 
have a higher probability 
(approximately one-and-one-half times) 
that fourth graders will score below the 
mean in standardized reading 
achievement. These relationships were 
even stronger among high-poverty 
schools. Students in schools with higher 
poverty levels were even more likely to 
score below the mean in reading 
achievement. 

The results of the logistic regression 
models also demonstrate that well- 
implemented Small Learning 
Communities appear to provide good 
learning environments. Small Learning 
Communities were better implemented 
in safer schools, in schools where there 
was a greater sense of Teacher 
Professional Community, and fewer 
Obstacles to Student Learning. Results 
showed that schools with higher 
concentrations of poverty but with well- 
developed Small Learning Communities 
were more likely to have fewer 
Obstacles to Student Learning and a 
greater sense of Teacher Professional 
Community as compared to the total 
fourth grade sample. Small Learning 
Communities had a greater positive 
impact on Obstacles to Student 
Learning and Teacher Professional 
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Community, the higher the 
concentration of poverty in schools. 

These findings support our hypothesis 
that there is a link between teachers' 
perceptions of school characteristics, 
teachers' sense of efficacy and 
collaboration, and students' school 
performance. Small Learning 
Communities were found to 
significantly relate to both school 
characteristics (i.e.. School Safety, 
Obstacles to Student Learning) and 
teacher characteristics (i.e.. Teacher 
Professional Community) when the 
school and teacher characteristics were 
related to students' achievement scores 
at the school level. Based on this 
evidence, it can be argued that Small 
Learning Communities indirectly relate 
to student achievement, because their 
existence increases the likelihood of 
having safer schools, schools with fewer 
Obstacles to Student Learning, and a 
greater sense of Teacher Professional 
Community. These factors relate to 
student achievement through our HLM 
and logistic regression analyses. 

Finally, school size (as measured by 
school enrollment) and teacher 
experience were shown to have some 
relationship to both Obstacles to 
Student Learning and teacher-reported 
Professional Community. Small schools 
with more experienced teachers were 
more likely to have fewer instructional 
obstacles and a better-developed sense 
of professional community. 



LIMITATIONS OF THE 
STUDY 

While the quantitative analysis of 
Children Achieving has contributed to a 
better understanding of how to 
improve educational systems, some 
limitations are worth noting. First, while 
the study used a broad spectrum of 
factors thought to relate to school 
performance, it is clear that there may 
be other relevant factors also, and 
hence worthy of consideration. For 
example, research has suggested that 
the home environment^^ is an important 
variable to consider when examining 
relationships between at-risk status and 
school performance. 

Similarly, while epidemiological studies 
have been noted as valuable 
contributions to the literature, 
identifying factors thought to protect a 
child from risk of poor school 
performance are needed. Cicchetti and 
Lynch,^'’ for example, argue that 
understanding the ways protective 



^ E.F. Dubow and M.F. Ippolito, "Effects of poverty 
and quality of the home environment on changes in 
the academic and behavioral adjustment of 
elementary school-age children." Journal of Clinical 
Child Psychology 23 (1994), pp. 401-412. J. Crane, 
"Effects of home environment, SES, and maternal 
test scores on mathematics achievement." Journal of 
Educational Psychology 89 (1996), pp. 306-314. 

” S.L Buka and LP. Lipsitt, "Toward a 
developmental epidemiological." In S.L Friedman 
and H.C. Haywood (Eds.), Developmental follow-up: 
Concepts, domains, and methods. San Diego: 
Academic Press, 1994. 

** D. Cicchetti and M. Lynch, "Toward an 
ecological/transactional model of community 
violence and child maltreatment: Consequences for 
children's development." Psychiatry: Interpersonal 
and Biological Processes 56 (1993), pp. 96-118. 
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factors can influence development is 
critical to best supporting children's 
needs. While some factors in the study 
could also be interpreted as protective 
factors, understanding others would be 
a valuable extension to extant research, 
such as parent involvement, enrollment 
in after-school enrichment programs, 
learning behaviors, and participation in 
school activities.^^ 

Lastly, it is worth noting that the design 
of the Children Achieving evaluation 
did not have a specifically identified 
control group. Because Children 
Achieving was a systemic reform that 
was implemented District-wide, analysis 
designs looked at variation across 
schools to assess the degree and 
quality of program success. 

CONTRIBUTIONS, 
REMAINING 
QUESTIONS, AND 
FUTURE DIRECTIONS 

The quantitative analysis of Children 
Achieving has produced several 
contributions to our understanding of 
systemic reform. First, this investigation 
empirically documents — in a reliable 
and valid way — ^the existence of a multi- 
faceted reform. This provides the basis 
for further research on how specific 
school reform components relate to 
student achievement and school 
performance. Without a solid 
understanding of the latent structure of 
the reform, identifying relationships 



” E.G. Epps, "Race, class, and educational 
opportunity: Trends in the sociology of education." 
Sociological Forum 10 (1995), pp. 593-608. 



between the reform and various 
outcomes (i.e., student achievement) 
would be compromised. 

Second, identifying additional school- 
level and community-level factors — in 
addition to individual 
characteristics — ^that potentially relate 
to student achievement, and 
understanding how these school 
characteristics, student characteristics, 
school outcomes, and school reform 
variables simultaneously relate, directly 
informs the design of effective services. 
For example, recent literature has 
suggested that a person's psychological 
sense of community, and other 
CO mm unity- 1 eve I factors such as social 
cohesiveness could be related to 
children's performance in school.^* To 
date, however, few studies have 
examined these questions in a manner 
that addresses the complex ways in 
which schools and teachers affect 
student school performance. 

Similarly, this investigation is 
population-based, and focuses on an 
especially vulnerable group of children. 
With the consequences of educational 
failure being pronounced for children in 



Brodsky, O'Campo, and Aronson, "'PSOC in 
community context: Multi-level correlates of a 
measure of psychological sense of community in low- 
income, urban neighborhoods." Journal of 
Community Psychology!? (1999), pp. 659-679. C.J. 
Coulton, J.G. Hopps, and R.H. Morris (Eds.), Social 
work at the m/7/enn/um: Critical reflections on the 
future of the profession (pp. 175-206). New York: 

The Free Press, 2000. S.M. Low, "The edge and the 
center: Gated communities and the discourse of 
urban fear." Amer/can Anthropologist 103 (2001), 
pp. 45-58. K.S. Newman, "Local caring: Social capital 
and social responsibility in New York's minority 
neighborhoods." 
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large urban centers/^ information on 
such children is critical. In addition, the 
majority of studies in the literature 
utilize small sample sizes, which limits 
the type of statistical analyses that can 
be used, compromises the power and 
rigor of statistical findings, and limits 
the ability to generalize results. Utilizing 
a population-based sample and 
providing replication analyses yield 
results that can be considered 
generalizable to elementary school- 
aged students in similar large urban 
environments. Furthermore, 
documenting that certain school 
characteristics and reform components 
demonstrate discernable relationships 
with children's performance in school 
regardless of a child's life circumstances 
highlights the notion that schools' 
structures and characteristics can be 
shaped to better serve their students. 

Finally, the current study serves as a 
valuable addition to the literature 
because it applies multivariate methods 
to the examination of relationships. By 
utilizing multiple logistic regression, the 
import of each risk factor on each 
school outcome was simultaneously 
assessed having controlled for the 
potential role of other important risk 
factors. Multivariate methods afford a 
more comprehensive and parsimonious 
understanding of the complex 
relationship among factors in a child's 
life and how those factors influence a 
child's performance in school, as well as 
how different parts of a school system 
relate to one other. 



REMAINING QUESTIONS 
AND FUTURE DIRECTIONS 

With multiple years of data about a 
school district, there is a considerable 
amount of promising future research to 
be conducted. Some of the following 
projects could be pursued. 

• Extend the current investigation to 
middle and high school years. How 
are results similar? How are they 
different? 

• Extend the current results to include 
student achievement in mathematics 
and science. How are results similar? 
How are they different? What trends 
emerge? Does improved reading 
predict subsequent improvements in 
other subjects? 

• Extend current results to track a 
cohort of students over multiple 
years, or a , cohort study across 
elementary, middle, and high 
school. How do children's 
achievement scores change over 
time as a function of the reform and 
other teacher/school characteristics? 



” National Center for Education Statistics, The 
condition of education 1996. 




3 zr 
O 



An Analysis of the Effect of Children Achieving on Student Achievement in Philadelphia Elementary Schools 



27 



APPENDIX A. 
ANALYSIS OF THE 
SURVEY OF 
PHILADELPHIA 
TEACHERS 

INTRODUCTION 

As part of the evaluation of Children 
Achieving — ^the School District of 
Philadelphia's comprehensive reform 
initiative funded in part by the 
Annenberg Foundation — ^the 
Consortium for Policy Research in 
Education and its partners, particularly 
the OMG Center for Collaborative 
Learning, developed, piloted, 
administered, and analyzed a survey of 
Philadelphia public school teachers. The 
survey utilized in the analyses for this 
report was administered in June 1999. 
The following describes the technical 
and methodological background to the 
survey. 

DEVELOPMENT OF THE 
SURVEY 

Development of the survey began in 
early 1996. Members of the survey 
design team — which included members 
from the Philadelphia Federation of 
Teachers (PFT) and the School District 
of Philadelphia, among others — 
reviewed and revised items gathered 
from other surveys of public school 
teachers. New items were developed as 
necessary to reflect the uniqueness of 
the Philadelphia context and of the 
Children Achieving initiative. Mark 
Smylie, from the Consortium for 



Chicago School Research, served as 
design consultant for the group. 
Teachers representing Philadelphia's 
elementary, middle, and high schools 
piloted the survey and their changes 
and comments were incorporated into 
the final version. 

RESPONDENTS 

In 1999, over 12,000 teachers and 
professional staff were asked to 
participate in completing the survey. 
PFT Building Representatives were 
enlisted to distribute the surveys, 
collect completed surveys, and return 
them to the OMG Center for 
Collaborative Learning. Most school 
principals offered teachers time during 
end-of-the-year staff meetings for 
completion of the survey. A letter 
jointly signed by the Superintendent, 
David Hornbeck, and the PFT 
President, Ted Kirsch, accompanied the 
survey and lent importance to the 
completion of the survey. 

The table below outlines the response 
rates from teachers by grade level. 

Several efforts were made to contact 
the PFT Building Representatives who 
did not return a survey packet for their 
school. Phone calls to both home and 
school, by members of the PFT and 
OMG staff, were not successful. 

DATA QUALITY CONTROL 

The surveys were printed on machine- 
readable forms, and were scanned, 
reviewed, corrected, and re-scanned in 
August 1999. 



28 



An Analysis of the Effect of Children Achieving on Student Achievement in Philadelphia Elementary Schools 



TABLE A.I. RESPONSE RATES AND 
NUMBERS TO THE CPRE TEACHER 
SURVEY, 1998-1999 





Percent 

Response 


N Response 


Elementary 


66 


3,254 


Middle 


54 


852 


High 


58 


1,224 


TOTAL 


63 


5,330 



ERRORS IN SURVEYS 

Despite efforts to avoid them, every 
survey has errors. There are three types 
of error that generally occur in survey 
research: sampling, non-response, and 
human error. Estimates of the amount 
of error and its effects on the results of 
the survey are dependent on the 
sample size, that is, the number of 
respondents to the survey. In general, 
the effects of error decrease as the 
number of respondents increases. 

Non-response bias is likely to result 
from non-participating schools and 
teachers who did not respond to the 
survey. Estimating how non-responding 
teachers and school differ from those 
that did respond is difficult. Basic 
employee demographic data helps us 
examine the difference between 
teachers in schools that participated in 
the survey and those who did not. 
Comparing data measuring 
race/ethnicity, gender, and teacher 
experience and education level, there 
were no significant differences between 
our sample and the teacher population 
as a whole in the School District. 

Within-school non-response is harder to 
evaluate. Since the distribution of the 
survey was random and anonymous, we 
have no way of knowing which teachers 



within a school responded to the survey 
and which did not. In our effort to 
understand the effects of error on the 
results of the survey, we can only 
assume in both cases that it is likely that 
non-responding schools and teachers 
are somehow different from those that 
did respond, and keep that caveat in 
mind as we interpret the results of the 
survey. 

DATA ANALYSIS 

A series of detailed steps were 
executed to investigate the reliability 
and construct validity of the survey. 
Specifically, these steps were followed 
in an effort to clarify and understand 
the inherent meaningful factors that 
compose the teacher survey. 

DEFINING VARIABLES 

Each of the independent variables were 
collected from all Philadelphia teachers 
in census surveys administered in 1997 
and 1999, as well as from administrative 
record data regularly collected by the 
School District. 

The following list of criteria was used to 
decide which variables in each group to 
keep as a representative of the 
category: 

• Each school has a measure of this 
variable. 

• This variable has the highest 
correlation with the dependent 
variable among other similar 
variables. 
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• This variable is operationalized in a 
way that maximizes its relationships 
with the dependent variable. 

• This variable would be expected to 
predict achievement based on 
previous research and improvement 
planning. 

• School scores for each of the scales 
were created by averaging 
individual respondents. 

DATA PREPARATION 

In this stage, the sample was screened 
for missing data on all questions being 
analyzed. In cases where there was 
missing data, an imputation method 
was utilized using Ward's multiple 
regression coefficient, which makes the 
best prediction of particular response 
given each of the other responses to 
that particular item. 

ITEM RELIABILITY 

Second, the internal consistency of an 
alpha program was run on each of the 
scales to assess the internal consistency 



of the measure and to conduct item 
analysis. Item total correlations were 
examined as well as skewness and 
kurtosis levels of each item in an effort 
to detect those items whose descriptive 
nature was significant discrepant from 
normal. 

CONSTRUCT VALIDITY 

In order to establish and provide 
evidence for construct validity of the 
survey, a series of exploratory and 
confirmatory factor analysis procedures 
were executed. These procedures are 
outlined in detail in the body of the 
report. 

ITEM CONVERSION 

In a final step, each factor was 
subjected to an area conversion 
process with a mean of 50 and a 
standard deviation of 15. This step 
allows for the comparison of students 
from different schools. All analyses 
were then conducted on the area- 
converted standard scores. 
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APPENDIX B. 1998-1999 TEACHER 
SURVEY— ITEM 0-BASED FACTOR 
DESCRIPTIONS 

TABLE B.1. QUALITY OF SCHOOL ENVIRONMENT— QUALITY OF SCHOOL EXPERIENCE 



1999 Philadelphia Teacher Survey: Items Loading on this Factor 

• My students feel safe coming to and going from this school. 

• I feel safe coming to and going from this school. 

• I feel safe in the building during the day. 

• Guns or weapons are a problem in this school. 

• Crime is a problem in this school. 

• The level of student misbehavior in this school interfered with my teaching. 

• Students are generally well behaved in the classroom. 

• I feel respected by students at this school. 

• Student behavior has changed in the past two years. 

• How students get along with each other has changed over the past two years. 

• How teachers get along with students has changed over the past two years. 

• Student attendance has changed in the past two years. 

• Sense of community in the school has changed in the past two years. 

• The quality of the Curriculum, Assessment, and Accountability has changed over the past two 
years. 

• The quality of instruction has changed over the past two years. 

• The quality of student academic performance has changed over the past two years. 

• My commitment to the school has changed over the past two years. 

• My teaching effectiveness has changed over the past two years. 

• My job satisfaction has changed over the past two years. 



TABLE B.2. TEACHER PROFESSIONAL COMMUNITY 



1999 Philadelphia Teacher Survey: Items Loading on this Factor 

• At this school, teachers work together to do "what is best for kids." 

• Teachers in this school share and discuss student work with other teachers. 

• Teachers talk about instruction in the teachers' lounge, faculty meetings, etc. 

• Teachers respect those who are trying new instructional approaches. 

• Teachers work together to design instructional programs that can be used in their classrooms. 

• Teachers at this school make a conscious effort to coordinate their teaching with instruction at 

other grade levels. 

• Teachers at this school respect those colleagues who are an expert at their craft. 

• Teachers at this school make a conscious effort to coordinate their teaching with others at the 

same grade levels. 

• I help maintain discipline in the entire school, not just my classroom. 

• I take responsibility for improving the school. 

• I feel responsible to help teachers do their best. 

• I feel responsible that all students learn. 

• Teachers try to understand parents' problems and concerns. 

• Teachers work closely with parents about how they can help their children. 

• Teachers in this school have high expectations for student learning. 




39 



32 An Analysis of the Effect of Children Achieving on Student Achievement in Philadelphia Elementary Schools 



TABLE B,3, SCHOOL SAFETY 



1999 Philadelphia Teacher Survey: Items Loading on this Factor 

• I am part of an environment that applies disciplinary rules more consistently in the school. 

• I am part of an environment that insists that all students wear picture identification badges. 

• I am part of an environment that creates alternative programs for students who consistently obey 

school rules. 

• I increase parents' presence in the school. 

• I am part of an environment that increases the number of school security personnel. 

• I help to create small learning communities to strengthen the relationship between teachers and 
students. 



TABLE B.4. OBSTACLES TO STUDENT LEARNING 



1999 Philadelphia Teacher Survey: Items Loading on this Factor 

• Students have a lack of basic skills. 

• Students have a lack of motivation. 

• There are too few alternative academic supports for students. 

• There is high student mobility in and out this school. 

• There is poor student attendance. 

• There is a lack of parental involvement in this school. 

• There is an inability to access community supports (i.e., health or mental health services) in this 
school. 

• There is not enough additional help in this school (i.e., classroom assistants, co-teachers, parent 
volunteers). 

• There is a great varying ability of students in my class. 

• Students lack appropriate student and work habits in my class. 
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TABLE B.5. CURRICULUM, ASSESSMENT, AND ACCOUNTABILITY— 
CHILDREN ACHIEVING REFORM 



1999 Philadelphia Teacher Survey: Items Loading on this Factor 

• The professional development experiences sponsored by my cluster and school this year have helped 
me prepare my students for the SAT-9. 

• The "Philadelphia Cluster Frameworks": 

• Are helpful in planning lessons for my class. 

• Provide me with sufficient detail about what students need to know to meet district standards. 

• Have led me to make significant changes in what I teach. 

• Have led me to change my teaching methods. 

• Set forth developmentally appropriate expectations for my students. 

• Have triggered a discussion in my school about what we teach. 

• Have helped improve the integration of subject matter. 

• Have helped me identify content areas in which I should increase my understanding. 

• Were accompanied by the resources necessary to implement them. 

• Explain the order or sequence of the subject matter to be taught. 

• My school's performance index score is a reasonable reflection of the school's progress. 

• I have made changes in my teaching strategies to help our school achieve this year's performance index 
target. 

• My school has adequate instructional resources to achieve this year's index target. 

• It is fair to hold teachers responsible for student achievement. 

• The Professional Responsibility system causes teachers to increase their efforts to improve learning. 

• I believe that the SAT-9 is a good measure of the knowledge and skills reflected in the district's content 
standards. 

• I believe that the SAT-9 is a good measure of the knowledge and skills my students need. 

• I have the Curriculum, Assessment, and Accountability materials I need to enable my students to do well 
on the SAT-9. 

• I believe that the SAT-9 has caused teachers to focus their instruction on important subject matter. 

• The SAT-9 is well aligned with the subject matter I teach in my class. 
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TABLE B.6. SMALL LEARNING COMMUNITIES (SLC)— CH/LDREN ACHIEVING REFORM 



1999 Philadelphia Teacher Survey: Items Loading on this Factor 

• My SLC has a clear theme that shapes my Curriculum, Assessment, and Accountability and 
instruction. 

• The majority of SLC meeting time is devoted to issues about Curriculum, Assessment, and 
Accountability and instruction. 

• The faculty assigned to my SLC have provided common planning time. 

• My SLC has well-defined space or location within the school. 

• The teachers in my SLC make decisions about Curriculum, Assessment, and Accountability and 
instruction 

• Students in my SLC have access to all the curricular options offered by my school. 

• I am involved in making decisions about the SLC budget. 

• I am involved in reviewing student work with other teachers. 

• I plan lessons with members of my SLC. 

• I have co-taught classes. 

• I have observed others' classes. 

• I have developed and shared assessment tools and practices. 

• I have identified individual intervention strategies for students who needed additional assistance. 

• I have re-grouped students for instructional purposes. 

• I have met with all SLC faculty members. 

• I have reviewed curricula for alignment with district standards. 

• I have met with parents to address student needs. 

• I have undertaken projects that involved all students and faculty in my SLC. 



TABLE B.7. TEACHING AND LEARNING NETWORK (TLN)— 
PROFESSIONAL DEVELOPMENT, CHILDREN ACHIEVING REFORM 



1999 Philadelphia Teacher Survey: Items Loading on this Factor 

• The TLN has been sustained and focused as opposed to short-lived and unrelated. 

• The TLN has deepened my understand of subject matter. 

• The TLN has deepened my understanding of how students learn subject matter. 

• The TLN has led me to make changes in my teaching. 

• The TLN staff have provided me with useful Curriculum, Assessment, and Accountability resources. 

• The TLN staff have provided me with useful assessment tools. 

• The TLN staff have the expertise to help me implement the standards and Curriculum, 

Assessment, and Accountability frameworks in my classroom. 

• The services offered by the TLN meet my needs as a teacher. 

• The TLN staff in my cluster have the knowledge and skills they need to help me improve my 

classroom practice. 
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TABLE B.8. TEACHER PRACTICE— CH/LDREW ACHIEVING REFORM 



1999 Philadelphia Teacher Survey: Items Loading on this Factor 

Estimate the change this year in how often you asked your students to... 

• Conduct research or collect data [about mathematics]. 

• Create charts, graphs, or tables. 

• Discuss different ways to solve math problems. 

• Write about math concepts or about how to solve math problems. 

• Write their own math problems. 

• Work in pairs or small groups [in math class]. 

• Work on a [math] group project. 

• Work on an individual [math] project. 

• Conduct research or collect data [about reading, English, or language arts]. 

• Critique other students' work [reading, English, or language arts]. 

• Discuss something they read. 

• Give oral reports. 

• Play educational games. 

• Read aloud. 

• Read silently to themselves. 

• Write about something they read. 

• Work in pairs or small groups [reading, English, or language arts]. 

• Work on a [reading, English, or language arts] group project. 

• Work on an [reading, English, or language arts] individual project. 
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APPENDIX C. RATIONALE FOR AND 
DESCRIPTION OF VARIABLE CATEGORIES 

1. All dichotomous variables were appropriately coded 1 and 0, respectively to 
reflect a risk and non-risk group. 

2. The frequency distributions of all continuous variables were also examined. 

Here, logical cut-off points were established based on the distribution, quintiles, 
and school-level average information. Based on these criteria, the following 
variables were dichotomized at the 40* percentile: 

• School Safety (40* percentile cut-off = 44.71) 

• Instructional Obstacles to Learning (40* percentile cut-off = 45.68) 

• Teacher Professional Community (40* percentile cut-off = 48.00) 

• Poverty Concentration (40* percentile cut-off = .851) 

3. The frequency distributions of teacher demographic continuous variables were 
also examined, logical cut-off points were established based on the distribution, 
and categories with the largest number were identified as the reference group 
for all continuous variables. 
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TABLE C.1 . DESCRIPTION OF VARIABLE CATEGORIES 


Variable 


Categorization Description 


Teacher Demographics 


Amount of education 


1 = Bachelor's Degree 

2 = Master's Degree 

3 = Master's + additional education 


Certification status 


1 = certification in subject teaching 
0 = no certification 


Number of years teaching in the district 


1 = year teacher 

2 = 2-5 years teaching 

3 = 6-10 years teaching 

4 = 11-15 years teaching 

5 = >15 years teaching 


School Characteristics 


Poverty 


1 = Extreme poverty (80 percent or more of students on 
free or reduced lunch) 

0 = non-extreme poverty 


School Safety 


1 = safer schools 
0 = less safe school 


School Obstacles to Student Learning 


1 = less obstacles 
0 = more obstacles 


Teacher Professional Community 


1 = more reported community 
0 = less reported community 


Small Learning Communities 


1 = more reported small learning communities 
0 = less reported small learning communities 
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